Stanford researchers testing RAG-based legal AI tools built by LexisNexis and Thomson Reuters, companies with billions in resources and a direct commercial incentive to get this right, found hallucination rates of 17 to 33 percent. That's the number worth sitting with before assuming "we added RAG" solved the hallucination problem. RAG reduces hallucination substantially, real research puts the reduction in the 40 to 71 percent range compared to a model with no retrieval at all, but reduces is not eliminates, and treating RAG as a hallucination fix rather than a hallucination mitigation is exactly how a well-funded, well-built product still ends up giving confidently wrong answers a third of the time.
Air Canada's chatbot invented a bereavement fare policy that never existed, and a tribunal later forced the airline to honor it anyway, since the company was held responsible for what its own AI told a customer. That's not a hypothetical risk. It's what happens when hallucination meets a real customer interaction with real consequences attached.
RAG reducing hallucination by 40 to 71 percent is a real, well-supported figure across multiple studies. It's also compatible with a RAG system still hallucinating on a meaningful share of real questions, particularly in complex reasoning or open-domain factual recall, where rates can still exceed 33 percent even with retrieval in place. Both facts are true simultaneously, and mistaking "RAG is better than no RAG" for "RAG is reliable" is the gap that produces incidents like Air Canada's.
Cause 1: retrieval issues
The most common cause is also the most mechanically simple: the retriever fetches a document that's topically related to the query but factually wrong, outdated, or subtly misleading, and if retrieval isn't well-tuned, that bad document propagates straight into the generated answer as if it were solid ground truth. This is distinct from the model "making something up" in the traditional sense, the model is doing exactly what it's supposed to do, generating an answer grounded in what it retrieved. The retrieval step itself is where the actual error was introduced.
The fix: this is the same retrieval-quality work covered in RAG chunking and pipeline design generally, better chunking, hybrid search, and reranking all reduce the rate at which a topically-close-but-wrong document gets retrieved in the first place. Specific to hallucination prevention, add explicit staleness filtering, a retrieved document that's outdated relative to a more recent one on the same topic should be down-weighted or excluded, not treated as equally valid ground truth just because it matched the query semantically.
Cause 2: fusion problems
This is the subtler, harder-to-catch failure mode. Even when every individual document the retriever pulls is accurate, the generation step can synthesize information across multiple documents in a way that produces an incorrect conclusion, "fusing" true premise A and true premise B into false conclusion C. This is genuinely difficult to catch with simple fact-checking against source documents, since each individual fact used in the answer really did come from a real, accurate source, the error lives in how they were combined, not in any single retrieved fact.
The fix: constrain the generation step to make its sourcing explicit, requiring the model to attribute specific claims to specific retrieved chunks rather than producing a single synthesized paragraph with no traceable origin for each statement. This doesn't just help a human reviewer catch fusion errors after the fact, it also measurably reduces the rate at which they occur in the first place, since a model prompted to cite its source per claim has less room to blend information freely across documents into something neither document actually said. Chain-of-thought prompting, having the model reason through its answer step by step before committing to a final response, is a genuinely effective complementary technique here, it gives the fusion step somewhere visible to go wrong instead of happening invisibly inside a single generation pass.
Cause 3: confidence misalignment
This is the failure mode that turns a retrieval or fusion error into a real, damaging incident rather than a quietly wrong answer nobody trusts. Language models generate output with consistently high apparent confidence regardless of whether the underlying claim is actually true, there's no natural, built-in signal distinguishing "I'm certain of this" from "I'm extrapolating past what my sources actually support." A hallucinated answer reads exactly as fluent and assured as a correct one, which is precisely what makes RAG hallucinations more dangerous than an obviously broken system, users have no reason to doubt an answer that sounds this certain.
The fix: this needs to be addressed as its own layer, separate from retrieval and generation quality. Explicitly prompt and train the system to signal uncertainty when retrieved context doesn't clearly support a confident answer, and to abstain, saying "I don't have enough information to answer that confidently" rather than generating a plausible-sounding guess. This is a real trade-off worth being honest about: a system tuned to abstain more often will also decline to answer some genuinely answerable questions, and tuning this balance correctly, confident enough to be useful, cautious enough to avoid fabrication, is an ongoing calibration exercise, not a one-time setting.
A detection technique worth knowing: contextual relevance scoring
Standard hallucination-detection metrics sometimes get this wrong in the other direction, flagging a response as a hallucination simply because it doesn't closely match the retrieved context's exact wording, even when the response is contextually accurate. Academic research on this problem (the NMISS approach, developed and tested specifically on healthcare RAG applications) addresses this by scoring for contextual relevance rather than surface-level text overlap, correctly distinguishing a genuinely fabricated claim from an accurate answer that was simply phrased differently than the source material. This matters practically: an evaluation pipeline that's too strict about exact wording match will generate false-positive hallucination flags, wasting review time on answers that were actually fine, while a pipeline that's too loose will miss real fabrications. Getting this calibration right in your own evaluation setup is worth real attention, not just picking a metric off the shelf and trusting its default threshold.
Matching verification effort to actual risk
Not every RAG deployment needs the same level of hallucination-prevention investment, and over-engineering a low-stakes internal tool wastes effort that would matter more elsewhere.
Low-stakes, internal, easily correctable (an internal FAQ bot, a documentation search assistant): standard RAG with reasonable retrieval quality is often sufficient, the cost of an occasional wrong answer is low and easily corrected by a human who already has context.
Customer-facing, moderate-stakes (a support chatbot, a sales assistant): add explicit source citation per claim and uncertainty signaling, so a wrong answer at least surfaces as flagged or attributable rather than delivered with unwarranted confidence.
High-stakes, regulated, or legally binding (healthcare guidance, legal information, financial advice, anything resembling Air Canada's fare-policy situation): add a verification layer, a separate check step that validates generated claims against source documents before the answer reaches the user, human-in-the-loop review for anything genuinely consequential, and explicit abstention as the default when confidence is uncertain rather than an edge case.
Regulatory pressure is real and growing here too, the EU AI Act's provisions taking effect through 2026 create real compliance obligations around transparency and factual reliability for AI systems serving European users, which is worth factoring into the verification-effort decision for any regulated or cross-border deployment, not just a risk-tolerance judgment call.
The bottom line
RAG hallucination has three distinct, separately addressable causes, bad retrieval, faulty fusion across accurate sources, and unwarranted confidence, and treating them as one problem with one fix is why "we added RAG" so often turns out to be an incomplete answer. Match your verification effort to actual stakes, a low-risk internal tool doesn't need the same rigor as anything resembling Air Canada's fare-policy chatbot, and remember that even well-resourced, professionally built systems still measure real, meaningful hallucination rates. The honest goal is managed risk, not zero hallucination, and building as if zero were achievable is its own kind of failure.
If your RAG chatbot's hallucination rate is a real concern, Flowagenz can audit which of these three causes is actually driving your specific errors before recommending a fix. Happy to walk through your setup on a short call.
Let's create something together
Get in touch with us today.
